Image Quilting

To implement Image Quilting, I take in a sample texture and a patchsize (as well as a tol and k val for picking levels of accuracy). I then calculate the overlap to be the floor of patchsize / 6. After that, I begin by randomly sampling a patch for the top left corner of the output image. Then, I iteratively add new patches in rows down the image. The patches overlap the previous patch by overlap pixels in the horizontal direction (within the same row) and in the vertical direction (once we go to the next row). To pick a new patch, I first grab the overlapping region from the current image. Then, I iterate through every pixel in the sample image and calculate the ssd of the patch starting at that pixel offset with the current overlapping region. Then, to choose the sample, I provide a couple of metrics. If tol is set (to some value slightly larger than one), the algorithm will pick a random patch with cost less than (tol*mincost), where mincost is equal to max(default_min_cost, cost[0]) where cost is the sorted list of costs of all patches ssd'd with the current patch, and default_min_cost is the lowest possible min cost, which can be set by the user or by default is zero. If tol is not set, the algorithm will pick a random sample from the k-lowest cost patches. k can be set by the user, or it is set to 10 by deafult. Finally, I calculate the optimal seam cut between the overlapping region of the current image and the chosen new patch, and then place the new patch into the image. I iteratively do this until the entire image is filled.

I implemented my own version of seam cuts for this project. It is partially based on the seam carving algorithm I wrote for the seam carving project, but also performs other functions such as image sewing and handling double seam cuts (i.e. both horizontal and vertical). For my energy function, I used the ssd of pixels. I constructed my energy image and map, and found the minimal seam, just as I did in seam carving. For the seam cut, however, I sampled the values from the first image if they were to the left of (or top of for vertical seams) or on the seam, and from the second image if they were to the right of (or bottom of for vertical seams) the seam. I experimented with taking the average of the two images for the values on the seam, but found it to simply make the image look blurry. For a double seam cut, I calculated both the optimal vertical and horizontal seams according to the overlapping regions in both directions. Then, I filled in pixels that were either on the left of the vertical seam or on the top of the horizontal seam as image 1, and the bottom right to be image 2. Implementing this was quite a challenge as it involved handling of 9 different regions of the output patch (considering that seams are not linear). Although the details of the implementation are uninteresting, it was a significant debugging challenge to make sure to get right. Below is a visual output of the process.

Texture transfer is a relatively easy extension of texture synthesis where, instead of synthesizing new textures, a target image is used as a template for the shape of the output, but a new texture is applied to it. To implement this, I added a second term to my cost function, which measures the ssd between the grayscale intensity of the target image's patch at the current region and the grayscale intensity of the proposed sample patch (or the gaussian filtered version of these results, which is often better to reduce the impact of noise). Then, when running the texture synthesis algorithm, I apply it to every location in the target image and calculate the optimal patch (or k patches or patches within the tol, depending on user input) to place according to a combination of this new cost and the original cost of the overlapping regions (from the texture synthesis section). I weight these two cost terms by an alpha term, which can be inputted by the user (is 0.5 by default).